Lay Perceptions of Scientific Findings:
The Risks of Variability and Lack of Consensus

Insert first author name here1, Insert second author name here1


1 Insert affiliation here

Introduction

The credibility of scientific research is in doubt, among lay consumer (Hornsey & Fielding, 2017) and scientist (Pashler & Wagenmakers, 2021) alike. Several tools have been proposed to combat this “crisis of confidence” (Ibid., p. 528). One such tool is the crowd science approach: “the organization of scientific research in open and collaborative projects” (Franzoni & Sauermann, 2014, p. 1). We focus on crowdsourced data analysis, also known as the many analysts or multi-analyst approach: giving the same dataset to different teams of scientists, who independently analyze it to answer the same research question and/or estimate a parameter of interest.
\(~~~~~\) According to science reformers, crowd-scientific findings that tell a consistent story should garner more confidence in the conclusions and increase public faith in science (Silberzahn et al., 2018; Uhlmann et al., 2019). Here, we ask if we can find empirical evidence for these claims: Does crowdsourcing data analysis improve lay perceptions of scientific findings?

Objectives

We explore the effects of scientific findings emerging from a crowd of researchers (vs. a typical research collaboration) on lay consumers’ posterior beliefs, perceptions of credibility, confidence in an aggregate effect size estimate, and ratings of researcher bias, error, and discretion.
\(~~~~~\) We compare the effects of providing lay consumers with a single, aggregate parameter estimate (the single-analyst condition) vs. multiple parameter estimates that (a) vary slightly and are all positive, leading to the same qualitative conclusion (the multi-consistent condition) or (b) vary widely and are of both signs, leading to differing qualitative conclusions (the multi-inconsistent condition). In all three conditions, the given estimates average to 5%.

Preregistered Hypotheses

Table 1: Predicted direction of effects

Measure Multi-consistent Multi-inconsistent
1. Posterior beliefs
2. Credibility
3. Confidence
4. Bias
5. Error
6. Discretion No prediction No prediction

Note. Table 1 indicates the predicted direction of effects for all outcomes, compared to the single-analyst condition and controlling for prior beliefs (a green plus/red minus indicates a positive/negative prediction, respectively). For example, we hypothesized that, compared to a single-analyst study and controlling for prior beliefs, ratings of credibility would be greater in the multi-consistent condition and lower in the multi-inconsistent condition.

Methods

We run an experiment (N = 1,498) with three conditions
\(~\) Single-analyst
A single, aggregate parameter estimate
\(~\) Multi-consistent
Multiple parameter estimates with low variance and high consensus
\(~\) Multi-inconsistent
Multiple parameter estimates with high variance and low consensus

Experimental Design

Results

Figure 1: Estimates (and 95% CIs) for all outcomes In line with our hypotheses, lay consumers of multi-analyst studies with inconsistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Find the results less credible
\(~~~\) Have less confidence in the average effect size estimate
\(~~~\) Believe the results are more likely to stem from bias
\(~~~\) Believe the results are more likely to stem from error

Contrary to our hypotheses, lay consumers of multi-analyst studies with consistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Believe the results are more likely to stem from error

We found no significant effects on
\(~~~\) Credibility of the results
\(~~~\) Confidence in the effect size estimate
\(~~~\) Ratings of bias

Exploratory results

For the additional exploratory measure, lay consumers of multi-analyst studies (both with consistent and inconsistent results)
\(~~~\) Perceive greater researcher degrees of freedom

Figure 2: Distribution of prior and posterior beliefs by condition

Discussion

Conclusion
\(~\) Crowdsourced data analysis has many worthy uses, but…
\(~~\) Variability and lack of consensus may evoke negative responses

Future Directions
\(~~~\) Perceptions of scientists?
\(~~\) Science communication and communicating uncertainty
\(~~~~\) Other suggestions?

Open Science Statement
The preregistration, survey materials, data, and code that support the findings of this study are openly available on GitHub and the OSF.
\(~~\) [Insert GitHub link here]
\(~~\) [Insert OSF link here]